Modern electronic systems rely on rapid execution of user programs and manipulation of data. The majority of distributed cluster computing is based on dataflow programming models. Hadoop™ and Spark™ are representative examples of platforms for distributed cluster computing. One feature in the dataflow programming model is that data and worker mappings are predefined, allowing the execution to be controlled by a cluster manager. Moreover, Hadoop™ and Spark™ aggressively cache data to utilize data locally. Such caching systems are usually based on software caching mechanisms that use large amounts of memory to contain input data and intermediate data.
In manipulating the data, the master controller initiates a shuffle of data between nodes of the cluster. The shuffle causes an instant bottle-neck of data attempting to traverse to the next processing location in the cluster. This is because the existing shuffle engine is centralized and synchronized. Each synchronized shuffle creates a barrier to parallel programming and execution. Therefore, the shuffle of the data can delay execution and cause large power spikes during the movement of the data. This spike in activity can cause over-investment in power and switching resources as well as delays in the execution of the user program.
Thus, a need still remains for electronic system with data exchange mechanism to improve execution reliability and performance in clustered computing environments. In view of the ever-increasing commercial competitive pressures, along with growing consumer expectations and the diminishing opportunities for meaningful product differentiation in the marketplace, it is increasingly critical that answers be found to these problems. Additionally, the need to reduce costs, improve efficiencies and performance, and meet competitive pressures adds an even greater urgency to the critical necessity for finding answers to these problems.
Solutions to these problems have been long sought but prior developments have not taught or suggested any solutions and, thus, solutions to these problems have long eluded those skilled in the art.